Tailored Application-specific System Call Tables

The system call interface defines the services an operating system kernel provides to user space programs. An operating system usually provides a uniform system call interface to all user programs, while in practice no programs utilize the whole set of the system calls. Existing system call based sandboxing and intrusion detection systems focus on confining program behavior using sophisticated finite state or pushdown automaton models. However, these automata generally incur high false positives when modeling program behavior such as signal handling and multithreading, and the runtime overhead is usually significantly high. We propose to use a stateless model, a whitelist of system calls needed by the target program. Due to the simplicity we are able to construct the model via static analysis on the program’s binary with much higher precision that incurs few false positives. We argue that this model is not “trivial” as stated by Wagner and Dean. We have validated this hypothesis on a set of common benign benchmark programs against a set of real-world shellcode, and shown that this simple model is, instead, very effective in preventing exploits. The model, encoded as an per-process tailored system call table, incurs virtually no runtime overhead, and should be practical to be deployed to enhance application and system security.